Practical notes on implementing derivatives

For many applications in graphics, one is confronted with the task of solving an optimization problem or solving a nonlinear system of equations. Efficient methods for solving these problems require derivatives to be available. Differentiation in numerical methods also occurs for many other reasons, such as computing forces from potential energy or computing normal and curvature information from an analytic level set. For most authors in the graphics community, the process of working out, implementing, and testing derivative routines does not seem to cause real difficulties. This is not always true, however, and some papers punt on the task in a variety of ways. One common way of avoiding the task of computing derivatives by hand is to rely on autodifferentiation. Autodifferentiation is a very reasonable and effective solution in some circumstances, which has received a fair amount of attention. This approach can be quite effective for rapid prototyping or for circumstances where computing derivatives does not take a significant fraction of the total runtime. I use autodifferentiation myself for a variety of purposes. There are many contexts in which autodifferentiation may fail to be adequate, such as being too slow or not sufficiently numerically robust. Autodifferentiation also does not work well in some circumstances, such as differentiating quantities depending on a singular value decomposition, a problem that occurs when working with constitutive models. In these cases, it is best to perform the differentiation by hand. Instead of doing this, some authors avoid the derivatives by using BFGS or L-BFGS instead of Newton’s method. Others approximate the derivatives, such as by omitting terms. Other authors choose to compute derivatives using Maple, Mathematica, or other similar software. These practices are all generally bad. BFGS and L-BFGS do not converge as rapidly as Newton’s method, especially for large sparse systems. Nor do numerical methods perform as well when derivatives are approximated. Derivatives computed symbolically using a software package tend to have many problems, including being enormously complicated, difficult to maintain, not numerically robust, and inefficient. In this document, I explore practical strategies for implementing derivatives in code. I focus our attention on strategies that are effective even when the derivatives appear hopelessly difficult to work out and implement. As such, many of the suggestions that I make are overkill for simple tasks.